Facial mirroring in virtual and augmented reality

ABSTRACT

A facial expression feedback system for providing real-time feedback on how closely a user matches his or her facial expression to a target expression is presented. The facial expression feedback system receives sensor data captured by a set of sensors. The facial expression feedback system evaluates the captured sensor data to generate a set of confidence scores with respect to a target facial expression. The system presents the feedback by modifying a multimedia presentation based on the generated set of confidence scores.

BACKGROUND Technical Field

The present disclosure generally relates to computing devices, and moreparticularly, to computing devices that perform facial recognition.

Description of the Related Art

Facial recognition is a biometric software application capable ofuniquely identifying or verifying a person by comparing and analyzingpatterns based on the person's facial contours. Facial recognition ismostly used for security purposes, though there is increasing interestin other areas of use, such as law enforcement, education, medicine,entertainment, as well as other enterprises.

SUMMARY

Some embodiments of the disclosure provide a facial expression feedbacksystem for capturing facial expression using sensor data and providingreal-time feedback on how closely a user matches his or her facialexpression to a target expression. The sensor data used for facialrecognition may include electromyography (EMG) data, optical data, audiodata, or a combination of different types of sensor data. In someembodiments, the facial expression feedback system evaluates thecaptured sensor data to generate a set of confidence scores with respectto a target facial expression. The system presents the feedback bymodifying a multimedia presentation based on the generated set ofconfidence scores. Examples of the multimedia presentation can be avideo game, a virtual reality (VR) presentation, an augmented reality(AR) presentation, or another form of visual/audio presentation.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the disclosure. It is not meant to be anintroduction or overview of all inventive subject matter disclosed inthis document. The Detailed Description that follows and the Drawingsthat are referred to in the Detailed Description will further describethe embodiments described in the Summary as well as other embodiments.Accordingly, to understand all the embodiments described by thisdocument, a Summary, Detailed Description and the Drawings are provided.Moreover, the claimed subject matter is not to be limited by theillustrative details in the Summary, Detailed Description, and theDrawings, but rather is to be defined by the appended claims, becausethe claimed subject matter can be embodied in other specific formswithout departing from the spirit of the subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are of illustrative embodiments. They do not illustrate allembodiments. Other embodiments may be used in addition or instead.Details that may be apparent or unnecessary may be omitted to save spaceor for more effective illustration. Some embodiments may be practicedwith additional components or steps and/or without all of the componentsor steps that are illustrated. When the same numeral appears indifferent drawings, it refers to the same or like components or steps.

FIG. 1 conceptually illustrates a facial expression feedback system,consistent with an exemplary embodiment.

FIG. 2 illustrates the facial expression processing device in greaterdetail, consistent with an exemplary embodiment.

FIGS. 3a-c conceptually illustrate an example AR presentation generatedby the facial expression feedback system.

FIG. 4 conceptually illustrates a process for providing facialexpression feedback, consistent with an exemplary embodiment.

FIG. 5 shows a block diagram of the components of a data processingsystem in accordance with an illustrative embodiment of the presentdisclosure.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent that the presentteachings may be practiced without such details. In other instances,well-known methods, procedures, components, and/or circuitry have beendescribed at a relatively high-level, without detail, in order to avoidunnecessarily obscuring aspects of the present teachings.

People with Autism Spectrum Disorder (ASD) have difficulty displayingfacial expressions and identifying facial expressions of others.Teaching facial mirroring has proven difficult because giving accuratereal-time feedback is difficult with a human instructor operating alone.

Some embodiments of the disclosure provide a facial expression feedbacksystem for capturing facial expression using sensor data and providingreal-time feedback on how closely a user matches his or her facialexpression to a target expression. The sensor data used for facialrecognition may include electromyography (EMG) data, optical data, audiodata, or a combination of different types of sensor data. In someembodiments, the facial expression feedback system evaluates thecaptured sensor data to generate a set of confidence scores with respectto a target facial expression. The system presents the feedback bymodifying a multimedia presentation based on the generated set ofconfidence scores. Examples of the multimedia presentation can be avideo game, a virtual reality (VR) presentation, an augmented reality(AR) presentation, or another form of visual/audio presentation.

FIG. 1 conceptually illustrates a facial expression feedback system 100,consistent with an exemplary embodiment. The facial expression feedbacksystem receives captured sensor data based on the facial expression of auser 105 and provides feedback regarding how closely the user's facialexpression resembles a target facial expression. As illustrated, thefeedback system 100 includes an EMG capture device 110, a video capturedevice 120, a facial expression processing device 130, and a feedbackpresentation device 140.

The EMG capture device 110, the video capture device 120, the facialexpression processing device 130, and the feedback presentation device140 may be individual electronic devices that are interconnected bycommunications mediums such as cables, wires, wireless signals, ornetworks. The devices 110-140 may also be components of the sameelectronic apparatus. An example computing device 500 that may implementthe facial expression system 100 will be described by reference to FIG.5 below.

The feedback system 100 may have one or more sensors for capturing thefacial expression of the user 105. In the example illustrated in FIG. 1,the EMG capture device 110 records the electrical activity produced byskeletal muscles of the user 105 through electrodes attached the face ofthe user. The video capture device 120 includes a camera for capturingthe video of the user's facial expression. The video capture device mayalso include a microphone for capturing any sound or voice that the usermay make. The EMG data captured by the EMG captured device 110 and thevideo data captured by the video capture device 120 are provided ascaptured sensor data 145 to the facial expression processing device 140.Though not illustrated, the feedback system 100 may include other typesof sensors for capturing other movements or sounds made by the user. Thecaptured sensor data 145 captures the facial expression of the user 105(among other things) such that the facial expression manifested in thecaptured sensor data can be referred to as the captured facialexpression (of the user 105).

The feedback system 100 processes the captured sensor data 145 at thefacial expression processing device 130. The facial expressionprocessing device 130 receives the captured sensor data and evaluatesthe captured sensor data with respect to a target facial expression. Theevaluation generates a score or a set of scores that quantify howclosely the user's facial expression (as manifested in the capturedsensor data) resembles the target facial expression. The generated scoreor set of scores is then used to generate a feedback presentation forthe user. In some embodiments, the sensor data is captured and processedin real-time to generate real-time feedback for the user 105. Thiscreates a feedback loop that allows the user to continually adjust hisor her facial expression to more closely match the desired expression.

As illustrated, the facial expression processing device 130 includes aset of facial expression detectors 150, a target specification module160, and a feedback presentation engine 170. In some embodiments, themodules 150-170 are modules of software instructions being executed byone or more processing units (e.g., a processor) of a computing device.In some embodiments, the modules 150-170 are modules of hardwarecircuits implemented by one or more integrated circuits (ICs) of anelectronic apparatus. An example computing device that may implement thefacial expression processing device 130 will be described by referenceto FIG. 5 below.

The feedback system 100 provides the generated feedback presentation atthe feedback presentation device 140. The feedback presentation device140 may be one display or a set of displays, a VR goggle, or any othertype of presentation device. The feedback presentation can be in theform of a video game, a 360-degree VR presentation, an AR presentationthat mixes computer generated graphics with real life images, or anotherform of visual/audio presentation. The feedback presentation may includeone or more visual objects and/or audio objects.

FIG. 2 illustrates the facial expression processing device 130 ingreater detail, consistent with an exemplary embodiment. As illustrated,the facial expression processing device 130 includes multiple facialexpression detectors 150 for different types of facial expressions(neutral, smile, frown, happy, sad, frightened, excited, etc.). Eachdetector of a particular facial expression receives the captured sensordata 145 and produces a set of confidence scores quantifying theresemblance of the captured facial expression to the particular facialexpression.

In some embodiments, the detectors 150 perform a 2-step process that (i)identifies which facial expression the user is showing based on thecaptured sensor data and (ii) extracts features from the captured sensordata to determine a set of intensity or confidence scores of theidentified facial expression. In some embodiments, each of the detectors150 is a model that is trained by machine learning to recognize aparticular type of facial expression based on one or multiple types ofsensor data. Such a machine learning model can be trained by usingtime-stamped sensor data (e.g., EMG data, camera footage, or both) withemotional labels. Training data can be manually coded (e.g., by humanwatching video playback and labeling expressions) or acquired withemotionally-coded visual stimuli (e.g., a jump scare in a horror videowould be labelled as eliciting a frightened expression).

The target specification module 160 specifies a target facial expression210 so that the facial expression processing device 130 knows which ofthe facial expression detectors 150 to apply for determining theconfidence scores from the captured sensor data. One or more facialexpression detectors are selected by target facial expression selection210, and the outputs of the selected detectors are collected asconfidence scores 220 and passed to the feedback presentation engine170. The target specification module 160 may provide a set ofcorresponding target confidence scores 230 for comparison with theconfidence scores 220.

In some embodiments, the target specification module 160 categorizes thecaptured facial expression of the user into one of the possible facialexpressions that the feedback system 100 is configured to detect. Thefacial expression that the captured facial expression is categorizedinto is then set to be the target facial expression. For example, thetarget specification module 160 may automatically select the targetfacial expression based on which of the trained facial expressiondetectors 150 outputs the highest scores for the captured facialexpression.

In some embodiments, the target specification module 160 receives thespecification for the target expression directly from a user through auser interface. An instructor may set the target facial expressionaccording to a lesson plan so the student may learn to match his or herown facial expression to the target facial expression.

The feedback presentation engine 170 receives the confidence score 220(i.e., the confidence score of the selected facial expression detector)and the corresponding target confidence score 230. The feedbackpresentation engine 170 determines the difference between the capturedfacial expression and the target expression by computing the differencebetween the received scores 220 and 230. This difference quantifies thediscrepancy between the captured facial expression of the user 105 andthe target facial expression.

In some embodiments, the feedback presentation engine 170 is amultimedia presentation engine that controls various visual or audibleobjects of a multimedia presentation. The feedback presentation engine170 presents the discrepancy between the target facial expression andthe captured facial expression by modifying the multimedia presentation.Specifically, the feedback presentation engine 170 modifies an existingobject in the multimedia presentation by a quantity that is determinedbased on the confidence scores 160.

Modifying the multimedia presentation based on the confidence scores 220may include using the difference between the confidence score 220 andthe corresponding target confidence score 230 to compute a discrepancyvalue by which the feedback presentation engine 170 modifies an existingvideo element or audio element in the multimedia presentation. Forexample, the feedback presentation engine 170 may modify an existingvisual element by changing its size, position, movement, color,saturation, or brightness according to the discrepancy value. Thefeedback presentation engine 170 may also modify an audio element in bymodifying its pitch or loudness according to the discrepancy value. Insome embodiments, the feedback presentation engine 170 interact with avideo game engine to modify the game play based on the set of confidencescores 220.

The multimedia presentation can be in the form of an augmented reality(AR) that includes a real-time image of a face, e.g., a real-timedisplay of the captured facial expression of the user 105. Themodification to the multimedia presentation can include overlaying orsuperimposing one or more visual markers on the real-time image of theface. Such visual markers may include arrows or highlights over featuresor areas of the face to give specific feedback (e.g., upward arrowtelling user to raise cheeks more).

FIGS. 3a-c conceptually illustrate an example AR presentation 300generated by the facial expression feedback system 100 (at the feedbackpresentation engine 170), in which visual markers superimposed over areal-time image of the user's face are used to give specific feedback.The figure shows the AR presentation at three different time instances.In each of the time instances, the real-time image of the user is shownin the AR presentation, and visual markers (arrows) are shown to providefeedback to the user regarding how to change his or her facialexpression. In the example, the target facial expression is set to“happy”.

FIG. 3a conceptually shows the AR presentation at a first time instancebefore the user has adjusted his facial expression. The feedback system100 generated a set of quantities 310 based on confidence scores thatare derived from evaluation of real-time sensor data with respect to“happy” as the target facial expression. The set of quantities 310correspond to different regions or features of the face. These featuresare labeled as A, B, C, D, E, which correspond to left eyebrow, righteyebrow, left cheek, right cheek, and lower lip (left/right defined fromthe perspective of the viewer since the image of the face is a mirrorimage), respectively. As illustrated, the quantity “+10” for the featureA corresponds to a downward arrow above the left eyebrow, while thequantity “−20” for the feature D corresponds to a upward arrow at theright cheek.

FIG. 3b shows the feedback system 100 updating the quantities A, B, C,D, and E based on the real-time sensor data captured near a second timeinstance. The newest real-time captured sensor data reflects the userchanging his facial expression based on the feedback presentation. Thevisual markers in the AR presentation are correspondingly modified. Asillustrated, the quantity ‘A’ has been updated to “0”, which correspondsto disappearance of the arrow above the left eyebrow. The quantity ‘D’has been updated to “−12”, which corresponds to a reduction in size ofthe arrow at the right cheek.

FIG. 3c shows the feedback system 100 further updating the quantities A,B, C, D, and E based on further real-time update of the captured sensordata, which reflect further changes in the facial expression of the usernear a third time instance. As illustrated, the quantities A, B, C, D,and E are all nearly zero. The feedback system 100 correspondinglymodifies the AR presentation 300 so that the arrows disappear,indicating that the user has successfully changed his facial expressionto match the target facial expression.

FIG. 4 conceptually illustrates a process 400 for providing facialexpression feedback, consistent with an exemplary embodiment. In someembodiments, one or more processing units (e.g., processor) of acomputing device implementing the facial expression processing device130 of the facial expression feedback system 100 performs the process400 by executing instructions stored in a computer readable medium.

The process 400 starts when the feedback system specifies (at 410) atarget facial expression. An instructor of a student using the facialexpression feedback system may set the target facial expression so thestudent may learn to match his or her own facial expression with theselected target facial expression. The feedback system may also specifythe target facial expression by categorizing the captured facialexpression of the user.

The feedback system receives (at 420) sensor data captured from a set ofsensors. The sensor data may include video data, audio data, and EMGdata that are captured based on the facial expression of the user. Thefacial expression of the user is therefore captured in the receivedsensor data.

The feedback system evaluates (at 430) the captured sensor data togenerate a set of confidence scores with respect to the target facialexpression. The feedback system may apply one or more detectors that areconfigured to detect the target facial expression to generate the set ofconfidence scores from the captured sensor data.

The feedback system then modifies (at 440) a multimedia presentationbased on the set of confidence scores. The feedback system controlsvarious visual or audible objects of the multimedia presentation.Existing objects in the multimedia presentation may be modified byquantities that are determined based on the set of confidence scores.The modification quantities may be calculated to track the discrepancybetween the captured facial expression and the target facial expression.The modification quantities may be calculated as the differences betweenthe set of confidence scores and a corresponding set of targetconfidence scores. The feedback system 100 may modify the visualappearance of an existing element in the multimedia presentation bychanging its size, position, movement, color, saturation, or brightness,etc. The feedback system 100 may also modify an audio element in themultimedia presentation by modifying the pitch or loudness of audiosignals. The feedback system 100 may also interact with a video gameengine to modify elements of the game play.

The feedback system presents (at 450) the modified multimediapresentation to the user as a feedback presentation so the user maylearn to match his or her own facial expression with the selected targetfacial expression. The process 400 may end or return to 420 tocontinuously receive sensor data and update the feedback presentation.

By analyzing real-time sensor data of the facial expression of the user,the facial expression feedback system 100 is able to provide a real-timefeedback to the user regarding his or her facial expression in aneasy-to-understand feedback presentation, enabling the user to adjusthis or her own facial expression to match a target facial expression inreal-time.

The present application may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present disclosure.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device. Computer readable programinstructions for carrying out operations of the present disclosure maybe assembler instructions, instruction-set-architecture (ISA)instructions, machine instructions, machine dependent instructions,microcode, firmware instructions, state-setting data, configuration datafor integrated circuitry, or either source code or object code writtenin any combination of one or more programming languages, including anobject oriented programming language such as Smalltalk, C++, or thelike, and procedural programming languages, such as the “C” programminglanguage or similar programming languages. The computer readable programinstructions may execute entirely on the user's computer, partly on theuser's computer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider). In some embodiments,electronic circuitry including, for example, programmable logiccircuitry, field-programmable gate arrays (FPGA), or programmable logicarrays (PLA) may execute the computer readable program instructions byutilizing state information of the computer readable programinstructions to personalize the electronic circuitry, in order toperform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions. These computer readable programinstructions may be provided to a processor of a computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks. The flowchart and block diagrams in the Figures (e.g., FIG. 4)illustrate the architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flowchart or block diagrams may represent amodule, segment, or portion of instructions, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). In some alternative implementations, the functions noted inthe blocks may occur out of the order noted in the Figures. For example,two blocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts or carry outcombinations of special purpose hardware and computer instructions.

FIG. 5 shows a block diagram of the components of data processingsystems 500 and 550 that may be used to implement the facial expressionfeedback system 100 or the facial expression process device 130 inaccordance with an illustrative embodiment of the present disclosure. Itshould be appreciated that FIG. 5 provides only an illustration of oneimplementation and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environments may be made based on designand implementation requirements.

Data processing systems 500 and 550 are representative of any electronicdevice capable of executing machine-readable program instructions. Dataprocessing systems 500 and 550 may be representative of a smart phone, acomputer system, PDA, or other electronic devices. Examples of computingsystems, environments, and/or configurations that may represented bydata processing systems 500 and 550 include, but are not limited to,personal computer systems, server computer systems, thin clients, thickclients, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, network PCs, minicomputer systems, anddistributed cloud computing environments that include any of the abovesystems or devices.

The data processing systems 500 and 550 may include a set of internalcomponents 500 and a set of external components 550 illustrated in FIG.5. The set of internal components 500 includes one or more processors520, one or more computer-readable RAMs 522 and one or morecomputer-readable ROMs 524 on one or more buses 526, and one or moreoperating systems 528 and one or more computer-readable tangible storagedevices 530. The one or more operating systems 528 and programs such asthe programs for executing the process 400 are stored on one or morecomputer-readable tangible storage devices 530 for execution by one ormore processors 520 via one or more RAMs 522 (which typically includecache memory). In the embodiment illustrated in FIG. 5, each of thecomputer-readable tangible storage devices 530 is a magnetic diskstorage device of an internal hard drive. Alternatively, each of thecomputer-readable tangible storage devices 530 is a semiconductorstorage device such as ROM 524, EPROM, flash memory or any othercomputer-readable tangible storage device that can store a computerprogram and digital information.

The set of internal components 500 also includes a R/W drive orinterface 532 to read from and write to one or more portablecomputer-readable tangible storage devices 586 such as a CD-ROM, DVD,memory stick, magnetic tape, magnetic disk, optical disk orsemiconductor storage device. The instructions for executing the process400 can be stored on one or more of the respective portablecomputer-readable tangible storage devices 586, read via the respectiveR/W drive or interface 532 and loaded into the respective hard drive530.

The set of internal components 500 may also include network adapters (orswitch port cards) or interfaces 536 such as a TCP/IP adapter cards,wireless Wi-Fi interface cards, or 3G or 4G wireless interface cards orother wired or wireless communication links. Instructions of processesor programs described above can be downloaded from an external computer(e.g., server) via a network (for example, the Internet, a local areanetwork or other, wide area network) and respective network adapters orinterfaces 536. From the network adapters (or switch port adaptors) orinterfaces 536, the instructions and data of the described programs orprocesses are loaded into the respective hard drive 530. The network maycomprise copper wires, optical fibers, wireless transmission, routers,firewalls, switches, gateway computers and/or edge servers.

The set of external components 550 can include a computer displaymonitor 570, a keyboard 580, and a computer mouse 584. The set ofexternal components 550 can also include touch screens, virtualkeyboards, touch pads, pointing devices, and other human interfacedevices. The set of internal components 500 also includes device drivers540 to interface to computer display monitor 570, keyboard 580 andcomputer mouse 584. The device drivers 540, R/W drive or interface 532and network adapter or interface 536 comprise hardware and software(stored in storage device 530 and/or ROM 524).

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computing device comprising: a processor; and astorage device storing a set of instructions, wherein an execution ofthe set of instructions by the processor configures the computing deviceto perform acts comprising: receiving sensor data captured by a set ofsensors, comrpising electromyography (EMG) data, optical data, and audiodata; evaluating the captured sensor data comprising the EMG, optical,and audio data to generate a set of confidence scores with respect to atarget facial expression; and modifying a multimedia presentation basedon the generated set of confidence scores.
 2. The computing device ofclaim 1, wherein the execution of the set of instructions furtherconfigures the computing device to perform an act, comprising: receivinga selection of the target facial expression from a plurality ofdifferent facial expressions.
 3. The computing device of claim 1,wherein generating the set of confidence scores comprises determining aset of discrepancies between the target facial expression and a capturedfacial expression based on the captured sensor data.
 4. The computingdevice of claim 1, wherein: generating the set of confidence scorescomprises applying a model to the captured sensor data, and the model istrained by machine learning to recognize the target facial expression.5. The computing device of claim 1, wherein modifying the multimediapresentation based on the generated set of confidence scores comprisesmodifying an existing object in a multimedia presentation by a quantitybased on the set of confidence scores.
 6. The computing device of claim1, wherein modifying the multimedia presentation comprises displaying avisual marker that points to a feature of a real-time image of a face inan augmented reality (AR).
 7. The computing device of claim 1, whereinmodifying the multimedia presentation comprises modifying an existingelement in a multimedia game.
 8. A computer program product comprising:one or more non-transitory computer-readable storage devices and programinstructions stored on at least one of the one or more non-transitorystorage devices, the program instructions executable by a processor, theprogram instructions comprising sets of instructions for: receivingsensor data captured by a set of sensors, comprising electromyography(EMG) data, optical data, and audio data; evaluating the captured sensordata copmrising the EMG, optical, and audio data to generate a set ofconfidence scores with respect to a target facial expression; andmodifying a multimedia presentation based on the generated set ofconfidence scores.
 9. The computer program product of claim 8, wherein:generating the set of confidence scores comprises applying a model tothe captured sensor data, and the model is trained by machine learningto recognize the target facial expression.
 10. The computer programproduct of claim 8, wherein modifying the multimedia presentation basedon the generated set of confidence scores comprises modifying anexisting object in a multimedia presentation by a quantity based on theset of confidence scores.
 11. The computer program product of claim 8,wherein modifying the multimedia presentation comprises displaying avisual marker that points to a feature of a real-time image of a face inan augmented reality (AR).
 12. A computer-implemented method comprising:receiving sensor data captured by a set of sensors, comprisingelectromyogaphy (EMG) data, optical data, and audio data; evaluating thecaptured sensor data comprising the EMG, optical and audio data togenerate a set of confidence scores with respect to a target facialexpression; and modifying a multimedia presentation based on thegenerated set of confidence scores.
 13. The computer-implemented methodof claim 12, further comprising receiving a selection of the targetfacial expression from a plurality of different facial expressions. 14.The computer-implemented method of claim 12, wherein generating the setof confidence scores comprises determining a set of discrepanciesbetween the target facial expression and a captured facial expressionbased on the captured sensor data.
 15. The computer-implemented methodof claim 12, wherein: generating the set of confidence scores comprisesapplying a model to the captured sensor data, and the model is trainedby machine learning to recognize the target facial expression.
 16. Thecomputer-implemented method of claim 12, wherein modifying themultimedia presentation based on the generated set of confidence scorescomprises modifying an existing object in a multimedia presentation by aquantity based on the set of confidence scores.
 17. Thecomputer-implemented method of claim 12, wherein modifying themultimedia presentation comprises displaying a visual marker that pointsto a feature of a real-time image of a face in an augmented reality(AR).
 18. The computer-implemented method of claim 12, wherein modifyingthe multimedia presentation comprises modifying an existing element in amultimedia game.